How do social scientists
answer questions using data?

• We then summarize these \(n\) observations by calculating a statistic, e.g. mean

  • All statistical procedures involve a statistic, very often sum or mean.
  • What are the properties of these sums and means?
  • Can the sample mean of age tell us anything about the population distribution of age?

• How do we know how far is this away from summary of all \(N\) (or even infinite number of!) units in population?

  • Key idea: Can we say something about distribution of statistics across (hypothetical) re-samples?
  • Key tool: Increase sample size to very large asymptotics

• The probability of \(\bar{X}_n\) being “far away” from \(\mu\) goes to \(0\) as \(n\) gets big
• Intuition: If we roll the six-sided die many times, what do you think the average of rolls will be?
• Important result: The distribution of sample means “collapses” to population mean if we draw many samples

• Intuition: Imagine you can collect many (large) samples and for each calculate a mean, resulting means form sampling distribution that has good properties

• Important result: We now know how far away \(\bar{X}_n\) can be from population mean of \(X_i\) (not \(\bar{X}_n\))!

• The Normal distribution is one of the most ubiquitous distributions in statistics

  • Follows “bell-shaped” curve
  • Mean and variance are two key characteristics (parameters) and are given in parentheses
  • When \(X\) is distributed normally, we write \(X \sim N ( \mu, \sigma^2 )\)

• Three key properties:

  • Unimodal: one peak at the mean
  • Symmetric around the mean
  • Everywhere positive: any real value can possibly occur

• Using properties of Normal distribution sample mean will be within \(2 \times \sigma^2 / n\) of the population mean 95% of the time

• We usually only have one sample, so we’ll only get one sample mean. So why do we care about LLN/CLT?

  • CLT gives us assurances our sample mean won’t be too far from population mean
  • CLT also helps us create measure of uncertainty for our estimates: standard deviation of the sampling distribution, or standard error (SE):

  \[ SE = \sqrt{\frac{\sigma^2}{n}} = \frac{\sigma}{\sqrt{n}} \]

• We will use data to guess something about the population distribution of \(X_i\) \(\Leftarrow\) statistical inference

• Examples of quantities of interest ( estimands ):

  • \(\mu = \mathbb{E} [ X_i ]\): the population mean (turnout rate in the population)
  • \(\sigma^2 = \mathrm{Var}[ X_i ]\): the population variance
  • \(\mu_1 - \mu_0 = \mathbb{E} [ X_i (1) ] - \mathbb{E} [ X_i (0) ]\): the population Average Treatment Effect (ATE)

• These are the things we want to learn about

• An estimate is one particular realization of the estimator

  • Ideally we’d like to know the estimation error (bias) \(\theta - \hat\theta\)
  • Problem: \(\theta\) is unknown
  • Solution: figure out the properties of \(\theta\) using probability

• \(\hat{\theta}\) is a random variable because it is a function of sequence of random draws \(\Rightarrow\) CLT and LLN apply!

• There are many (\(\infty\)) different possible estimators:

  • \(\hat{\theta} = \bar{X}_n\) : the sample proportion of respondents who support Biden
  • \(\hat{\theta} = X_1\) : just use the first observation
  • \(\hat{\theta} = \max( X_1 , ... , X_n )\) : pick the maximum of all observations
  • \(\hat{\theta} = 0.5\) : always guess 50% support

• How good are these different estimators?

• We usually rely on mean, partly because it makes LLN and CLT apply

• Remember: the sample mean is a random variable

  • Different samples give different sample means
  • Noise “bumps” sample mean away from population mean; We assume away bias
  • \(\bar{X}\) has a distribution across repeated samples–sampling distribution–that we know has nice properties

• UnBIASedness: Sample proportion is on average equal to the population proportion

• We can use a special rule to binary random variables to calculate standard deviation (SD):

\[\sqrt{\mathrm{Var}(\bar{X})} = \sqrt{\frac{p(1 − p)}{n}}\]

• Problem: we don’t know \(p\)!

• Solution: estimate the SE using sample mean

\[\sqrt{\widehat{\mathrm{Var}}(\bar{X})} = \sqrt{\frac{\bar{X}(1 − \bar{X})}{n}} \class{fragment}{= \sqrt{\frac{0.39 (1 − 0.39)}{1007}} \approx 0.0153}\]

• Awesome-r: Get a range of plausible values

• Confidence interval: way to construct an interval that will contain the true value in some fixed proportion of repeated samples

• Central Limit Theorem:

\[\bar{X} \sim N \left( \underbrace{\mathbb{E}[X_i]}_{p}, \underbrace{\frac{\mathrm{Var}(X_i)}{n}}_{\frac{p(1-p)}{n}} \right)\]

• Noise: \(\bar{X} − p\) is approximately normal with mean 0 and SD equal to \(\sqrt{\frac{p(1-p)}{n}}\)

• \(100 \times (1 − \alpha)\) % confidence interval: \(CI = X ± z_{\alpha/2} \times SE\)

  • In polling, \(\pm z_{\alpha/2} × SE\) is called the margin of error

• 90% CI: \[[0.39 − 1.64 × 0.0153, 0.39 + 1.64 × 0.0153] = [0.364, 0.415]\]
• 95% CI: \[[0.39 − 1.96 × 0.0153, 0.39 + 1.96 × 0.0153] = [0.360, 0.420]\]
• 99% CI: \[[0.39 − 2.58 × 0.0153, 0.39 + 2.58 × 0.0153] = [0.351, 0.429]\]

• Less confidence \(\rightarrow\) Wider intervals

• According to CLT the sample means are normally distributed with variance of \(\sigma^2 / n\) \(\Rightarrow\) use sample mean and sample size to calculate estimate of uncertainty (sampling variance)

• Come up with confidence level (usually 95%) and use mean and variance estimates to see where the population mean is likely to be if we would to re-sample many times \(\Rightarrow\) Calculate range of ~2 standard errors (square root of sampling variance) around the estimated population mean

• 95% confidence interval includes population mean in 95% of re-samples

• This is important if all relevant outcomes within sample are observed

  • e.g. sample mean, median, polarization, etc.

• But, for causal (“what-if”) question we cannot observe all relevant outcomes within sample \(\Rightarrow\) internal validity

  • Why?

• Does the minimum wage increase the unemployment rate?

  • Factual: Unemployment rate in US went up after the minimum wage increased
  • Counterfactual: Would it have gone up if the minimum wage increase not occurred?

• Does having a daughter affect a judge’s rulings in court?

  • Factual: A judge with a daughter gave a pro-choice ruling
  • Counterfactual: Would they have done that if they had a son instead?

• Fundamental problem of causal inference:

  • We can never observe counterfactuals \(\Leftarrow\) must be inferred or assumed

• Question: Did the first village have larger share spent on water sanitation because the head of the council was female?

• Concern: What if there are other differences between villages or the budget share leads to female council representation?

• Treatment (\(T_i = 1\)) group: Villages with female head of council

• Control (\(T_i = 0\)) group: Villages with male head of council

• Outcome variable, \(Y_i\): Share of spending

• Imagine two states of the world: one in which you receive some treatment and another in which you do not \(\Rightarrow\) potential outcomes

  • Treated, \(Y_i (1)\): spending on water sanitation if village \(i\) had a female head?
  • Untreated/Control, \(Y_i (0)\): spending on water sanitation if village \(i\) had a male head?

• Average Treatment Effect (ATE):

  \[ \frac{1}{n} \sum_{i = 1}^{n} Y_i (1) − \frac{1}{n} \sum_{i = 1}^{n} Y_i (0) = \frac{1}{n} \sum_{i = 1}^{n} \left[ Y_i (1) − Y_i (0) \right] \]

  • Difference between average treated and untreated potential outcomes
  • Does having female head of village lead to increase in spendings on water sanitation on average?
  • Note: You might come close to observing this treatment effect if you can observe the \(Y_i (0)\) the instant before an intervention and the \(Y_(1)\) the instant after, but strictly speaking, you are not observing them at the same time \(\Rightarrow\) Focus on ATE

• Fundamental problem of causal inference:

  • We only observe one of the two potential outcomes.
  • In terms of potential outcomes: observe \(Y_i = Y_i (1)\) if \(T_i = 1\) or \(Y_i = Y_i (0)\) if \(T_i = 0\)

• To infer causal effect, we need to infer the missing counterfactuals!

• Did village spend more on water sanitation because of female council head?

  • \(\rightarrow\) find a village that has male council head but very similar otherwise

• NJ increased the minimum wage. Causal effect on unemployment?

  • \(\rightarrow\) find a state similar to NJ that didn’t increase minimum wage

• Random assignment enables us to create two groups whose treated and untreated potential outcomes are the same in expectation

• The treatment group provides us with a random sample of \(Y_i (1)\), and the control group provides us with a random sample of \(Y_i (0)\)

• The difference-in-means estimator compares average outcomes between two samples: treatment and control group

\[ \text{Difference-in-means} = \bar{Y}_{\text{treated}} - \bar{Y}_{\text{untreated}} \]

2. Non-interference: A subject’s potential outcomes reflect only whether they receive the treatment themselves

   • A subject’s potential outcomes are unaffected by how the treatments happened to be allocated

3. Excludability: A subject’s potential outcomes respond only to the defined treatment, not other extraneous factors that may be correlated with treatment

   • Importance of defining the treatment precisely and maintaining symmetry between treatment and control groups (e.g., through blinding)

• Regression with covariates: Block possible known (?) confounders

• Matching: Use known (?) covariates to match units and then compare within matched sets

• Event study: Use before and after treatment comparison within the same unit

• Regression Discontinuity: Use some naturally occurring discontinuity (e.g. tax exemption threshold, borders, etc.) to compare units around it

• Differences-in-Differences: Compare trends between treated and untreated units even if we know there might be differences between them

• Many other!

• What do we substitute for ???? We need an educated guess …

• Steps:

  • Assume the true treatment effect is your difference in means \(\Rightarrow\) reconstruct schedule of potential outcomes
  • We can use approach similar to re-sampling for sampling mean
  • Re-shuffle the treatment, for each unit take the potential outcome corresponding to new treatment see what differences-in-means you could get

• Can also be done using parametric approach analogous to sample mean

\[ SE(\bar{Y}_{\text{treated}} - \bar{Y}_{\text{untreated}}) = \sqrt{\frac{\sigma_{\text{treated}}^2}{n_{\text{treated}}} + \frac{\sigma_{\text{untreated}}^2}{n_{\text{untreated}}}} \]

• We call these values of quantity of interest hypotheses and can see how likely are we to observe our result under those hypotheses to test them

  • Reconstruct schedule of potential outcomes assuming hypothesis is true (e.g. individual treatment effect is 0)
  • Re-shuffle the treatment, for each unit take the potential outcome corresponding to new treatment see what differences-in-means you could get
    
  • See how likely are you to observe value at least as extreme as yours \(\Rightarrow\) \(p\)-value